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REAL-TIME  DETECTOR  OF  HUMAN  FATIGUE 
Final  Technical  Report 
Detection  of  Human  Fatigue 


1  Introduction 

Fatigue  is  a  continuing  problem  for  modem  humans.  Lack  of  sleep  is  reported  to  be  the  greatest 
producer  of  such  feelings.  Objective,  especially  physiological  indicators  of  fatigue  are  being 
developed,  albeit  slowly.  The  most  common  indicator  is  impaired  performance  with  reaction 
time  as  one  of  the  major  measures.  As  people  become  fatigued  there  is  a  significant  increase  in 
the  frequency  with  which  longer  than  “normal”  reaction  times  occur.  As  fatigue  increases  there 
is  an  increase  in  the  frequency  with  which  such  signs  occur.  It  should  be  pointed  out  that  we  are 
not  dealing  with  a  general  slowing  of  responding;  rather  the  frequency  of  longer  than  normal 
response  latencies  increases.  Similarly,  the  likelihood  of  not  responding  to  signals  increases  as  a 
function  of  time  on  task. 

Since  sleep  deprivation  is  an  accepted  procedure  for  producing  feelings  of  fatigue,  much  of  the 
research  on  human  fatigue  has  focused  on  the  effect  of  sleep  deprivation  on  alterations  in 
performance.  It  is  well  known  that  measures  of  performance  decrement  do  not  occur,  or  occur 
only  minimally,  when  tasks  are  of  short  duration  as  well  as  when  rest  periods  are  allowed 
between  performance  of  different  tasks  or  segments  of  the  same  task.  The  Psychomotor 
Vigilance  Task  (PVT)  is  one  of  the  few  tasks  that  consistently  demonstrates  alterations  in 
performance  as  a  function  of  variables  such  as  sleep  deprivation  and  the  ingestion  of 
psychoactive  substances.  Duration  of  task  performance  to  demonstrate  such  effects  is  usually  10 
and  in  some  cases  20  minutes.  No  studies  have  been  conducted  using  longer  periods  of  task 
performance  to  determine  whether  performance  decrements  occur  as  a  function  of  Time  on  Task 
(ToT),  in  the  absence  of  sleep  deprivation  or  the  administration  of  psychoactive  agents. 

The  development  .of  physiological  measures  associated  with  feelings  of  “fatigue”,  especially 
measures  that  can  be  applied  in  operational  environments  has  been  slow.  To  be  usable  in 
operational  settings,  the  measurement  should  not  place  constraints  on  the  operator.  Thus 
measures  requiring  the  application  of  sensors  have  to  be  avoided.  Though 
electroencephalographic  measures  have  been  developed  that  detect  fatigue  effects,  the  use  of  this 
technique  requires  application  of  electrodes.  We  have  singled  out  two  physiological  measures 
involving  remote  monitoring  technology  and  one  measure  involving  a  behavioral  measure  that 
we  believe  to  have  a  high  potential  for  identifying  fatigue  effects.  These  are  video  based 
measures  of  oculomotor  activity,  Laser-Doppler  Vibrometry  for  the  recording  of  measures  such 
as  cardiovascular  and  muscle  activity  and  third,  a  behavioral  measure,  namely  instrumenting  the 
mouse  to  capture  slight  pressure  changes  that  may  indicate  anticipation  and  under  some 
circumstances  the  phenomenon  of  “motor  overflow”. 

We  developed  a  task  labeled  the  Enhanced  Psychomotor  Vigilance  Task  (EPVT)  with  the 
expectation  that  we  would  see  alterations  in  alertness  as  a  function  of  ToT.  Because  of  our 
interest  in  using  oculomotor  measures,  we  designed  our  task  so  that  gaze  shifts  had  to  be  made. 
We  also  incorporated  a  more  cognitive  requirement  by  adding  a  running  memory  task  to  the 
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EPVT.  Our  first  concern  was  with  demonstrating  that  this  task  produces  ToT  effects  on  response 
latency  and  in  concurrently  recording  a  number  of  bio-behavioral  measures  with  the  intent  of 
identifying  elements  associated  with  performance  impairment. 

2  Experimental  task 

The  psychomotor  vigilance  test  (PVT)  was  developed  by  Wilkinson  (1982)  and  improved  by 
Dinges  and  collaborators  (1991)  to  evaluate  performance  lapses.  It  is  sensitive  to  sleep 
deprivation  as  well  as  pharmacological  manipulations.  Central  nervous  system  depressants 
decrease  and  central  nervous  stimulants  improve  performance  on  this  test.  We  believe  that 
demonstrating  attentional  lapses  in  non-sleep  deprived  and  non-pharmacological  conditions 
requires  a  more  sensitive  task,  one  that  incorporates  perceptual  as  well  as  cognitive  demands.  We 
developed  a  task  called  the  enhanced  psychomotor  performance  task  (EPVT).  We  enhanced  the 
task  in  two  dimensions,  the  first  perceptual,  the  second  cognitive  in  nature.  The  perceptual 
manipulation  requires  subjects  to  not  only  attend  to  centrally  presented  information  (as  found  in 
the  PVT),  but  to  information  presented  at  3  locations,  a  central  one  and  10  degrees  to  the  left  and 
right  of  the  central  location.  Subjects  cannot  predict  the  location  of  stimulus  presentation.  A 
second  manipulation  requires  subjects  to  abstract  information  rapidly.  The  cognitive 
manipulation  requires  subjects  to  retain  information  (a  running  memory  task)  and  respond  after  a 
sequence  of  specific  events  has  transpired. 

The  EPVT,  unlike  the  PVT,  presents  the  operator  with  three  (rather  than  one)  count-up 
millisecond  timers.  Count-up  is  initiated  on  one  of  the  three  timers,  and  the  subject  is  required  to 
depress  the  left  mouse  button  to  stop  the  timer.  For  the  cognitive  component,  the  subject  is 
required  to  retain  information  about  the  nature  of  the  digit  in  the  millisecond  position  of  the 
timer.  She  is  required  to  depress  the  right  mouse  button  following  a  sequence  of  three  odd 
integers  at  the  millisecond  position.  She  thus  has  to  abstract  information,  retain  it  in  memory,  and 
finally  respond  after  a  specified  sequence  of  odd  integers  has  been  presented.  The  millisecond 
segment  of  the  timer  is  under  operator,  rather  than  subject  control.  We  thus  control,  without  the 
subject’s  awareness,  the  frequency  with  which  a  sequence  of  three  odd  integers  occurs.  Stimulus 
onset  asynchrony  is  variable,  averaging  2.5  seconds.  If  a  subject  does  not  respond  to  a  stimulus, 
the  next  stimulus  occurs  automatically  after  an  interval  of  4.0  sec.  The  subject’s  manual  response 
to  detecting  the  scrolling  timer  stops  the  timer  and  the  time  remains  on  the  screen  for  400  ms. 
The  timer  then  resets  to  the  default  value  of  0000  at  the  three  locations  where  stimuli  are 
presented. 

3  Data  collection  procedures 

Subjects  were  Washington  University  student  volunteers  and  were  paid  $30.00  for  participating 
in  this  experiment.  They  came  to  an  office  at  the  Washington  University  School  of  Medicine  and 
were  required  to  fill  out  a  number  of  forms  including  informed  consent.  Following  this,  they 
were  brought  to  the  laboratory  and  the  instrumentation  and  procedure  were  explained.  The 
camera  based  data  acquisition  system  was  calibrated.  This  took  on  the  average  30  seconds.  A 
piece  of  reflecting  tape  was  placed  over  the  site  from  which  cardiac  activity  was  to  be  recorded 
with  the  Laser-Doppler  Vibrometer  (LDV).  Subjects  then  read  the  instructions  for  the  EPVT  and 
were  provided  with  a  5-minute  practice  period.  The  experimenter  observed  responses  to  assure 
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her  that  the  subject  was  following  instructions.  Some  subjects  had  difficulty  in  understanding 
instructions  for  the  memory  task.  The  practice  period  gave  us  the  opportunity  to  further  instruct 
them  in  the  performance  of  this  task.  We  also  monitored  reaction  time  (RT)  to  the  simple 
reaction  time  (SRT)  task  and  prior  to  starting  the  experiment  proper  asked  subjects  to  attempt  to 
maintain  their  response  latencies  at  or  better  than  a  specified  level.  This  level  was  dependent  on 
their  response  latency  during  the  practice  task.  The  suggested  latency  was  identified  as  their 
modal  latency  plus  approximately  50  ms. 

We  collected  usable  data  for  20  subjects,  10  male,  10  female.  Data  for  5  subjects  was  discarded 
principally  because  of  the  unavailability  of  one  of  the  data  sets,  camera  based  data  or  Laser- 
Doppler  Vibrometry  data.  Data  loss  was  occasioned  in  one  case  by  the  subject  not  completing 
the  60  minutes  of  task  performance,  in  four  cases  because  of  major  movement  artifact  interfering 
with  our  ability  to  record  acceptable  signals. 
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4  Measures  utilized 

The  following  measures  were  abstracted  from  the  collected  data. 

4. 1  Simple  reaction  time  -  mouse  based  responses 

4.1.1  Time  on  Task  (ToT)  effects 

4. 1 .2  Long  latency  responses  -  their  temporal  distribution 

4. 1 .3  Short  latency  responses  -  their  temporal  distribution 

4. 1 .4  Missed  signals  and  interstimulus  responses 

4.  i  .5  Comparison  of  siinple  reaction  time  responses  using  the  mouse  and  pressure 
sensors 

4.2  Running  memory  task  performance 

4.2.1  Time  on  task  effects 

4.2.2  Accuracy  measures 

4.3  Camera  based  measures  associated  with  response  latency 

4.3.1  Saccadic  eye  movements  to  target  (tT)  and  return  from  target  (rT) 

4.3.2  Timing  of  saccadic  eye  movements  with  respect  to  response  enactment 

4.3.3  Head  movements  associated  with  saccades 

4.3.4  Pupil  diameter  measures  reflecting  performance  impairment 

4.3.5  Pupil  diameter  changes  associated  with  “long”,  “normal”  and  “short”  latency 
responses 

4.3.6  Pupil  diameter  changes  as  a  function  of  ToT 

4.3.7  Data  loss  considerations 

4.4  Cardiovascular  measures 

4.4. 1  Cardiovascular  measures  associated  with  “expectancy” 

4.4.2  Fractionation  of  heart  period  into;  pre-ejection;  left  ventricular  ejection,  and 
post  ejection  periods. 
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5  Results  obtained 

5. 1  Simple  reaction  time  -  mouse  based  responses 

5.1.1  Time  on  Task  (ToT)  effects 

Analysis  based  on  mean  reaction  time 

Table  1  presents  the  average  reaction  time  per  subject  summarized  in  5-minute  blocks.  Figure  1 
presents  the  average  reaction  time  across  20  subjects. 
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0.343 

0.361 

0.378 

0.416 

15-20 

0.350 

0.488 

0.356 

0.799 

0.392 

0.308 

0.375 

0.354 

0.401 

0.410 

,20-25 

0.351 

0.512 

0.413 

0.735 

0.368 

0.319 

0.355 

0.475 

0.412 
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30-35 
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0.402 
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0.325 

0.435 
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0.472 

0.425 
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0.392 
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0.409 

0.403 

0.338 
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0.518 

0.437 

40-45 

0.402 

0.577 

0.481 

0.404! 

0.394 

0.359 

0.629 

0.534 

0.663 

0.439 

45-50 

0.417 

0.644 

0.452 

0.422 

0.434 

0.421 

0.644 

0.615 

0.786 

0.480 

50-55 

0.450 

0.616 
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0.462 

0.419 

0.400 

0.439 
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0.356 

0.357 

0.490 

0.373 

0.300 

0.382 

0.418 

0.439 

0.319 

5-10 

0.366 

0.359 

0.362 

0.833 

0.394 

0.304 

0.369 

0.445 

0.562 

0.324 

10-15 

0.428 

0.358 

0.352 

1.582 

0.408 

0.305 

0.360 

0.485 

0.556 

0.328 

i  15-20 

0.496 

0.331 

0.363 

1.442 

0.451 

0.314 

0.424 

0.450 

0.638 

0.338 

120-25 

0.525 

0.344 

0.381 

1.127 

0.486 

0.317 

0.415 

0.462 

0.587 

0.346 

25-30 

0.602 

0.351 

0.379 

0.610 

0.457 

0.317 

0.407 

0.468 

0.475 

0.344 

30-35 

0.648 

0.371 1 

0.411 

1.047 

0.478 

0.314 

0.420 

0.436 

0.409 

0.352 

35-40 

0.612 

0.379 

0.502 

0.928 

0.752 

0.325 

0.421 

0.421 

0.448 

0.349 

40-45 

0.710 

0.371 ! 

0.504 

1.283 

0.576 

0.325 

0.434 

0.428 

0.475 

0.343 

45-50 

0.930 

0.357 

0.461 

1.016 

0.702 

0.320 

0.441 

0.453 

0.466 

0.358 

50-55 

0.749 

0.361 

0.523 

0.832 

0.957 

0.323 

0.499 

0.433 

0.502 

0.362 

55-60 

0.589 

0.372 

0.545 

0.445 

0.787 

0.336 

0.486 

0.435 

0.425 

0.362 

Table  1  -  Average  reaction  time  per  subject 
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Mean  Reaction  Time 


Figure  1  -  Average  reaction  time  across  subjects 

It  is  readily  apparent  from  Figure  1  that  subjects  demonstrate  an  increase  in  reaction  time  over 
the  initial  15  minutes  of  task  performance,  maintain  that  level  of  performance  for  the  next  20 
minutes  and  then  demonstrate  a  further  increase  in  average  reaction  time. 

Analysis  based  on  “normalized”  reaction  time  measures 

Because  of  the  large  differences  between  subjects  in  reaction  time,  the  data  was  “normalized”  by 
expressing  average  reaction  time  for  specified  periods  as  a  ratio  of  average  reaction  time  for  the 
initial  five  minutes  of  task  performance. 

We  presumed  that  the  first  five  minutes  of  task  performance  would  be  the  period  in  which  they 
were  most  highly  motivated  to  perform,  assuming  that  there  was  no  learning  involved  in  task 
performance.  We  thus  used  the  initial  5-minute  period  as  the  “criterion”  period  and  compared 
performance  for  successive  periods  with  the  initial  5-minute  period.  This  was  done  for  minutes 
5-10  and  then  in  10-minute  segments. 

Figure  2  presents  the  results  of  this  analysis  averaged  over  the  20  subjects. 
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Normalized  Mean  Reaction  Time 
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Figure  2  -  Normalized  reaction  time  across  subjects 

It  is  readily  apparent  that  most  subjects  had  the  shortest  average  RT  during  the  initial  5  minutes 
of  task  performance.  Three  subjects  performed  better  in  minutes  5-10  (F10,  F20,  M14)  than  the 
initial  5-minute  period  and  one  subject  (F12)  had  her  best  performance  in  minutes  10-20  and  20- 
30.  Thus  we  can  conclude  that  for  the  majority  of  subjects,  there  was  little  learning  involved  in 
performing  the  simple  reaction  time  component  of  this  task,  or  that  all  learning  was  successfully 
accomplished  in  the  initial  five  minute  training  period. 

Were  there  Time  on  Task  effects?  That  is,  did  subjects  demonstrate  an  increase  in  reaction  time 
as  a  function  of  ToT?  A  number  of  subjects  demonstrated  consistent  increases  in  the  ratio  as  a 
function  of  ToT.  (Under  consistent  increase  we  included  all  subjects  who  did  not  show  a  reversal 
in  the  ratio  for  more  than  one  successive  comparison.)  This  was  characteristic  of  50%  of  the 
subjects,  namely  F13,  F17,  F18,  F20,  F23,  M10,  Mil,  M12,  M17,  and M18. 

A  number  of  subjects  demonstrated  their  poorest  performance  not  during  the  last  10  minutes  but 
during  the  preceding  10  minutes  of  task  performance.  Subjects  F10,  M14,  M15,  M16,  and  M17 
fit  this  description. 

There  were  a  number  of  subjects  who  did  not  fit  the  pattern  of  increasing  reaction  time  as  a 
function  of  ToT.  Three  subjects  had  their  best  performance  in  the  last  30  minutes,  subjects  F21, 
F22,  and  M13.  Subject  F14  showed  the  least  consistent  pattern,  this  is  a  subject  where  we  could 
document  that  she  occasionally  fell  asleep  during  task  performance. 

Two  subjects  had  their  best  performance  not  during  the  initial  5-minute  period  but  later  in  task 
performance.  This  characterized  subjects  F12  and  M22.  The  latter  did  not  demonstrate  better 
performance  during  these  periods  but  performance  equal  to  that  seen  during  the  initial  5  minutes. 

In  general,  we  can  conclude  that  most  subjects  demonstrated  an  increase  in  response  latency  as  a 
function  of  ToT.  Fifteen  of  the  twenty  subjects  had  their  longest  average  reaction  times  during 
the  last  two  periods. 
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On  average  we  see  an  increase  in  reaction  time  from  the  initial  5-minute  period  through  minutes 
10-20,  asymptotic  performance  for  the  next  ten  minutes,  a  second  period  of  increase  in  reaction 
time  from  minutes  30  to  50  and  a  maintenance  at  that  level  for  the  last  10  minutes.  That  was  the 
average  pattern.  We  would  like  to  point  out  that  no  single  subject  demonstrated  such  a  pattern. 

We  also  evaluated  median  reaction  time  for  successive  periods  as  well  as  normalized  median 
reaction  times.  Results  of  these  analyses  are  shown  in  Figure  3.  These  results  deviate  somewhat 
from  the  analysis  based  on  means.  In  all  cases  the  average  based  on  means  is  longer  than  that 
based  on  medians.  That  difference  increases  as  a  function  of  time  on  task.  It  is  our  assumption 
that  these  differences  can  be  accounted  for,  in  great  part,  by  the  significant  increase  in  long 
latency  responses  over  time.  These  long  latency  responses  have  a  greater  effect  on  mean  than  on 
median  values. 


Median  Reaction  Time 


15  20  25  30  35  40  45  50  55  60 


Figure  3  -  Median  reaction  time  across  subjects 

5.1.2  Long  latency  responses  -  their  temporal  distribution 

Since  our  major  concern  is  with  the  identification  of  lapses  in  alertness  and  since  we 
hypothesized  that  there  should  be  an  increase  in  long  latency  responses  as  a  function  of  ToT,  we 
performed  the  following  analysis. 

We  identified  the  50  longest  latency  simple  reaction  time  (SRT)  responses  made  by  all  20 
subjects  assuming  that  long  latency  responses  are  most  likely  associated  with  brief  periods  of 
loss  of  alertness.  If  such  periods  increase  in  frequency  as  a  function  of  time  on  task,  the 
distribution  of  these  50  events  should  show  an  increase  in  frequency  of  occurrence  over  time. 
The  frequency  of  occurrence  of  these  events  in  consecutive  10- minute  periods  is  depicted  in 
Table  2  and  Figure  4. 


10 


Bio-Behavior 
Analysis  Systems 


-  -  Contractor:  Bio-Behavior  Analysis  Systems,  LLC 
C'oriff act  hio . FA9S50:O4:C-OCS62 '  : 

Project  Title:  Real-Time  Detector  of  Human  Fatigue 
Report  Title:  Final  Technical  Report:  Detection  of  Human 
Fatigue 

_ Date:  05  Aug  2005 _ 


Time 

(seconds) 

M10 

Mil 

M12 

M13 

M14 

M15 

M16 

M17 

M18 

M22 

0-600 

2 

jBI 

2 

mm 

5 

5 

0 

0 

0 

4 

600-1200 

2 

mm 

2 

— 

8 

1 

3 

1 

0 

2 

1200-1800 

6 

9 

5 

13 

5 

4 

2 

9 

1 

2 

1800-2400 

mm 

8 

9 

3 

8 

6 

7 

10 

0 

7 

2400-3000 

E3 

20 

0 

10 

20 

18 

16 

20 

15 

3000-3600 

13 

14 

12 

1 

14 

14 

20 

14 

29 

20 

Time 

(seconds) 

F10 

F12 

F13 

F17 

F18 

F21 

F22 

F23 

0-60C 

0 

4 

0 

4 

0 

3 

6r 

8  . 

.  1 

600-1200 

0 

5 

2 

19 

1 

6 

7 

13 

17 

3 

1200-1800 

3 

3 

3 

6 

3 

8 

7 

9 

10 

9 

1800-2400 

10 

14 

11 

7 

11 

4 

8 

6 

3 

9 

2400-3000 

21 

8 

15 

10 

11 

12 

5 

9 

5 

13 

3000-3600 

16 

16 

19 

4 

24 

14 

20 

WMBi 

7 

15 

Table  2  -  Distribution  of  long  latency  responses  per  subject 


Figure  4-  Distribution  of  long  latency  responses  as  a  function  of  ToT 
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Table  3  shows  the  mean  and  median  distribution  of  long  latency  responses  as  a  percentage  of 
long  latency  events  that  occur  in  successive  10-minute  periods. 


Time  period  (seconds) 

mean 

median 

0-600 

6.1 

5.0 

601-1200 

11.7 

6.0 

1201-1800 

11.7 

13.0 

1801-2400 

14.6 

16.0 

2401-3000 

26.2 

30.6 

3001-3600 

29.3 

28.0 

Table  3-  Distribution  of  mean  and  median  long  latency  responses 

A  Friedman  two-way  analysis  of  variance  was  performed.  The  Chi  squared  r  value  wasTO.O  and 
significantly  different  from  chance  at  the  p  <  0.005  level  indicating  that  the  highest  frequency  of 
long  latency  events  occurred  in  the  last  two  time  periods. 

We  should  point  out  that  the  pattern  of  increasing  number  of  long  duration  responses  as  a 
function  of  ToT  does  not  characterize  every  one  of  the  20  subjects.  Four  subjects  deviate 
considerably  from  this  pattern  (F14,  F21,  F22,  and  M13).  F14  has  been  previously  described  as 
the  subject  who  demonstrated  the  largest  number  and  durations  of  periods  of  non-responding, 
from  which  we  inferred  that  she  had  dozed  off  during  these  “lapses  in  performance”.  F21  and 
F22  demonstrate  no  consistent  change  in  average  reaction  time  across  the  successive  5-minute 
periods.  M13  had  the  slowest  reaction  times  of  all  subjects  (excepting  F14)  for  the  initial  30 
minutes  of  task  performance,  showed  a  marked  drop  in  reaction  time  in  the  next  interval,  and 
maintained  the  faster  reaction  times  for  the  remainder  of  the  hour. 

The  distribution  of  response  latencies  for  16  of  the  20  subjects  accords  with  our  expectation  that 
the  majority  of  long  latency  responses  occur  late  in  task  performance.  All  four  subjects  not 
demonstrating  this  pattern  (M13,  F14,  F21,  F34)  show  a  pattern  of  longer  latency  responses  early 
in  task  performance.  We  conclude  that  for  a  majority  of  subjects,  there  is  an  increase  in  long 
latency  responses  as  a  function  of  ToT  demonstrating  that  the  EPVT  is  sensitive  to  ToT  effects. 

5.1.3  Short  latency  responses  -  their  temporal  distribution 

Table  4  depicts  the  distribution  of  the  50  fastest  reaction  time  responses.  We  expected  that  this 
distribution  would  be  little  affected  by  ToT  considerations.  As  the  table  demonstrates,  there  is  a 
decrease  in  such  responses  as  a  function  of  ToT.  The  largest  number  occur  during  the  initial  5- 
minute  period,  dispelling  the  idea  that  there  might  be  a  learning  effect.  Whatever  learning 
occurred  transpired  during  the  practice  period  for  most  of  our  subjects. 
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EPVT  Distribution  of  50  Fastest 


F10 

FI  2 

FI  3 

FI  4 

FI  7 

FI  8 

F20 

F21 

F22 

F23 

1. 

600(0-10) 

34 

2 

10 

19 

20 

19 

19 

9 

7 

16 

2. 

1200(10-20) 

7 

11 

15 

0 

9 

8 

10 

5 

4 

10 

3. 

1800(20-30) 

2 

10 

13 

7 

6 

7 

10 

5 

6 

7 

4. 

2400(20-30) 

1 

3 

7 

2 

6 

4 

5 

10 

17 

6 

5. 

3000(30-40) 

1 

8 

4 

3 

4 

4 

6 

12 

10 

8 

6. 

3600(40-50) 

5 

16 

1 

19 

5 

8 

0 

9 

6 

3 

M10 

Mil 

M12 

M13 

M14 

M15 

M16 

M17 

M18 

M22 

1. 

600(0-10) 

16 

16 

14 

0 

10 

14 

6 

24 

26 

3 

2. 

1200(10-20) 

15 

7 

11 

0 

5 

15 

16 

16 

15 

7 

3. 

1800(20-30) 

6 

5  , 

13 

2 

8 

5 

13  . 

2 

6 

8 

4. 

2400(20-30) 

5 

9  "  . 

8 

18 

13 

9 

0 

4 

2 

6 

5. 

3000(30-40) 

7 

4 

1 

16 

6 

6 

11 

2 

1 

14 

6. 

3600(40-50) 

1 

9 

3 

14 

8 

1 

4 

2 

0 

12 

Table  4  -  Distribution  of  50  fastest  responses 


5.1.4  Missed  signals  and  interstimulus  responses. 

Missed  signals  are  the  ultimate  in  defining  periods  of  loss  of  alertness.  False  alarms  could  be 
divided  into  two  types.  The  first  involved  the  enactment  of  a  second  response  shortly  after  the 
response  that  stopped  the  timer.  Relatively  few  such  responses  were  obtained.  The  majority  of 
interstimulus  responses  could  be  characterized  as  “anticipatory”  responses.  We  defined  as 
anticipatory  any  response  that  occurred  in  the  second  half  of  the  interstimulus  interval. 

Table  5  depicts  the  occurrence  of  such  responses  grouped  in  5-minute  intervals.  Included  in  this 
analysis  are  both  anticipatory  responses  as  well  as  missed  events.  The  FA:M  ratios  in  the  table 
reflect  the  number  of  events  that  are  False  Alarms  as  well  as  Missed  stimuli  (no  response). 
Twelve  subjects  had  five  or  fewer  missed  events  over  the  60-minute  period;  five  subjects  had 
between  6  and  10  missed  events;  two  subjects  had  between  11  and  20  missed  events;  and  one 
subject  had  more  than  21  such  events.  The  subject  with  the  largest  number  of  such  events  was 
F14.  Once  again,  this  is  the  subject  who  demonstrated  obvious  periods  where  her  eyes  were 
closed  and  she  appeared  to  be  asleep.  She  contributed  52%  of  all  the  missed  signals.  Taking 
missed  signals  for  that  subject  out  of  the  picture,  there  are  three  others  who  have  a  relatively  high 
rate  of  missed  signals,  subjects  F17,  M16,  and  M17.  The  three  in  combination  contributed  63% 
of  the  remaining  missed  signals.  Since  these  4  subjects  were,  in  many  respects,  quite  deviant 
from  the  remaining  16,  we  also  calculated  SRT  accuracy  for  the  16  (more  homogeneous) 
remaining  subjects  There  is  a  marginally  significant  ToT  effect  for  the  10  minute  averaged  data 
including  all  20  subjects  F(5,l  14)  1.902,  p<0.099.  The  results  of  performing  the  same  analysis 
for  the  16  more  homogeneous  subjects  are  as  shown  in  Table  6. 
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Total  Mistakes  In  the  Simple  Reaction  Time  Task  Across  Time  (data  with  mouse  only) 

FI  0(20) 

F12(19) 

F13(18) 

FI  4(25) 

F17(19) 

FI  8(23) 

F20(22)  F21(18) 

F22(23) 

F23(21) 

300 

0 

0 

0 

1 

0 

0 

0 

0 

3 

1 

600 

0 

0 

0 

0 

1 

3 

0 

0 

3 

0 

900 

0 

0 

0 

3 

1 

1 

0 

1 

1 

0 

1200 

2 

2 

1 

26 

2 

0 

1 

0 

2 

0 

1500 

2 

1 

0 

21 

2 

2 

2 

0 

9 

0 

1800 

1 

0 

2 

1 

0 

3 

0 

1 

5 

0 

2100 

1 

1 

0 

9 

3 

0 

0 

4 

8 

2 

2400 

0 

1 

1 

3 

4 

1 

1 

1 

4 

0 

2700 

6 

0 

0 

17 

2 

1 

1 

1 

6 

1 

3000 

2 

1 

0 

11 

3 

0 

0 

0 

4 

0 

3300 

4 

1 

1 

5 

3 

2 

0 

0 

10 

1 

More 

2 

o. 

,,  0 

0 

•  .  7 

3 

1 

0 

'  5 

'  1 

Total 

20 

7 

5 

97 

28 

16 

6 

8 

60 

6 

FA:M  Ratio 

18:2 

7:0 

5:0 

4:93 

18:10 

13:3 

6:0 

7:1 

54:6 

6:0 

Ml  0(22) 

Ml  1(20) 

Ml  2(22) 

Ml  3(22) 

Ml  4(22) 

Ml  5(20) 

Ml  6(21)  Ml  7(23) 

Ml  8(21) 

M22(25) 

300 

2 

0 

2 

0 

1 

6 

1 

5 

0 

3 

600 

1 

5 

1 

6 

3 

1 

0 

1 

0 

0 

900 

1 

0 

4 

2 

8 

0 

1 

18 

0 

1 

1200 

0 

1 

4 

2 

3 

4 

0 

7 

1 

1 

1500 

0 

0 

2 

4 

6 

6 

13 

5 

2 

4 

1800 

1 

3 

1 

1 

6 

2 

3 

8 

0 

2 

2100 

0 

1 

3 

8 

10 

1 

6 

7 

1 

9 

2400 

1 

0 

4 

7 

12 

3 

6 

14 

1 

1 

2700 

4 

6 

0 

3 

15 

3 

5 

16 

0 

6 

3000 

2 

2 

2 

3 

8 

5 

5 

9 

1 

3 

3300 

6 

2 

3 

5 

14 

12 

5 

8 

0 

6 

More 

2 

4 

2 

4 

6 

7 

4 

20 

4 

5 

Total 

20 

24 

28 

45 

92 

50 

49 

118 

10 

41 

FA:M  Ratio 

19:1 

18:6 

27:1 

38:7 

83:9 

45:5 

38:11 

102:16 

8:2 

36:5 

Table  5  -  SRT  accuracy  in  5-minute  intervals 


Time  period 

Average  frequency 
(N=20) 

Average  frequency 
(N=16) 

0-600 

2.5 

2.56 

601-1200 

5.05 

2.69 

1201-1800 

6.05 

4.25 

1801-2400 

6.97 

5.44 

2401-3000 

7.70 

5.38 

3001-3600 

8.25 

7.06 

Table  6  -  Average  frequency  of  missed  signals 
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5.1.5  Comparison  of  simple  reaction  time  responses  using  the  mouse  and 
pressure  sensors 

This  analysis  was  conducted  on  the  subset  of  subjects  (N=5)  for  which  most  of  each  subject’s 
pressure  responses  could  be  recorded. 


Mouse-Pressure  Response  Time  Differences 

M10 

Mil 

M13 

F1 1 

F12 

Mean 

0.064 

0.055 

0.058 

0.059 

0.064 

Standard  Error 

0.002 

0.005 

0.004 

0.004 

0.003 

Median 

0.059 

0.061 

0.058 

0.062 

Mode 

0.064 

0.069 

0.061 

0.061 

0.059 

Standard  Deviation  - 

0.012 

0.021 

0.013 

0.021 

0.013 

Sample  Variance 

0.000 

0.000 

0.000 

0.000 

0.000 

Kurtosis 

-0.513 

0.855 

1.253 

2.167 

1.651 

Skewness 

0.162 

-0.820 

0.790 

0.822 

1.048 

Range 

0.045 

0.085 

0.077 

0.101 

0.055 

Minimum 

0.044 

0.007 

0.033 

0.020 

0.047 

Maximum 

0.089 

0.093 

0.110 

0.121 

0.101 

Sum 

1.729 

1.109 

1.450 

1.475 

1.605 

Count 

27.000 

20.000 

25.000 

25.000 

25.000 

Confidence  Level  (95.0%) 

0.005 

0.010 

0.007 

0.009 

0.005 

Table  7  -  Mouse-pressure  response  -  Time  differences 


As  shown  in  Table  7,  the  average  time  difference  between  reaction  time  as  measured  with  the 
mouse  and  as  measured  with  the  pressure  sensor  for  the  five  subjects  was  60  ms.  In  all  cases  the 
pressure  response  preceded  the  mouse  identified  response.  Thus  the  mouse  is  neither  an  accurate 
nor  a  reliable  tool  for  measuring  time  intervals.  A  number  of  research  papers  have  demonstrated 
similar  results,  the  discrepancy  varying  as  a  function  of  the  mouse  used.  When  subjects  were 
interviewed  after  task  performance,  they  occasionally  commented  on  the  fact  that  they  had  to 
press  the  mouse  switch  a  second  time  to  stop  the  timer.  This  observation  was  bom  out  by  the 
finding  that  one  could  occasionally  see  a  double  response  output  from  the  pressure  sensor  while 
the  mouse  output  occurred  concurrent  with  the  second  pressure  sensor  response. 

Though  “anticipatory”  responses  were  occasionally  obtained  with  the  mouse  switch,  and  an 
“anticipatory  response  is  defined  as  one  occurring  in  the  second  half  of  the  intertrial  interval, 
such  responses  were  much  more  frequently  seen  from  the  pressure  sensor.  In  addition  the 
pressure  sensor  picked  up  partial  pressure  on  the  mouse  prior  to  the  enactment  of  a  response. 
Whether  this  response  was  produced  simply  by  the  weight  of  the  finger  on  the  sensor  or  whether 
some  pressure  was  exerted  could  not  be  determined.  Future  studies  will  attempt  to  improve  on 
the  current  sensor  configuration  so  that  we  will  get  more  reliable  pressure  sensing  as  well  as 
calibrating  the  sensor  so  that  one  can  determine  whether  partial  pressure  was  exerted.  We  can  say 
that  for  four  of  the  five  subjects,  reaction  time  was  faster  when  the  finger  rested  on  the  mouse 
(reflected  in  an  output  from  the  pressure  sensor)  than  when  there  was  no  output  from  the  pressure 
sensor.  Observing  subjects  performing  the  task,  one  could  identify  times  when  the  finger  was  on 
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and  other  times  when  the  finger  was  off  the  sensor.  Systematic  observations  will  have  to  be  made 
to  verify  the  above  “impression”. 

We  thus  believe  that  improving  the  output  from  the  pressure  sensors  will  add  an  important 
dimension  to  not  only  the  accurate  measurement  of  reaction  time,  but  will  also  allow  us  to  more 
sensitively  monitor  “multiple  responses”,  anticipatory  responses,  and  partial  as  well  as 
“overflow”  responses. 

Below  are  a  few  examples  of  responses  captured  with  both  pressure  sensor  and  mouse  that 
reflect  the  additional  information  than  can  potentially  be  gathered  from  a  refined  analysis  of 
reaction  time  data. 

The  following  three  figures,  Figure  5,  Figure  6,  and  Figure  7  are  not  random  samples  but  were 
selected  to  demonstrate  some  of  the  measures  that  can  be  obtained  in  addition  to  RT  and 
response  duration. 

Data  was  sampled  at  17  millisecond  intervals  and  depicted  are  1000  consecutive  data  points  or 
17  seconds  of  data.  The  channels  are  identified,  the  top  three  deal  with  output  from  the  pressure 
sensors  located  on  the  left  and  right  mouse  buttons,  and  the  channel  labeled  Pressure  Sensor  Side 
samples  pressure  from  the  left  side  of  the  mouse,  the  usual  location  for  the  right  thumb.  The  next 
two  channels  depict  the  output  from  the  mouse,  identifying  both  onset  of  response  as  well  as 
response  termination.  The  mouse  left  channel  reflects  simple  RT  and  the  mouse  right  channel 
shows  responses  to  the  running  memory  task  (requiring  a  response  to  a  sequence  of  three  odd 
integers).  The  last  two  channels  depict  stimulus  information.  Stimulus  left  represents  stimuli 
located  to  the  left  of  center,  stimulus  right,  to  the  right  of  center,  and  when  both  stimulus  left  and 
right  are  activated  stimulus  information  is  available  at  the  central  location.  In  addition  to 
providing  information  about  stimulus  location,  these  two  channels  also  provide  information 
concerning  whether  a  given  stimulus  ends  in  an  odd  or  an  even  integer.  If  the  stimulus  bar  is 
narrow,  it  identifies  an  even  integer,  wide,  an  odd  integer. 

Figure  5  shows  information  sampled  between  274  and  291  seconds  following  the  initiation  of  the 
experiment.  Our  focus  here  is  on  the  left  pressure  sensor  response.  To  the  first  stimulus  (right) 
we  see  concurrent  responses  from  both  the  mouse  and  pressure  sensor.  Then  there  is  a  pressure 
sensor  response  not  reflected  in  mouse  activation.  The  next  stimulus,  also  at  the  right  side  of  the 
display,  demonstrates  anticipatory  activity  from  the  pressure  sensor  channel.  The  next  stimulus, 
at  the  central  location,  demonstrates  the  non-linearity  of  the  current  sensor  output.  Partial 
pressure  is  exerted  on  the  mouse  and  we  get  only  a  small  “blip”  from  the  pressure  sensor 
concurrent  with  the  mouse  response.  The  same  phenomenon  is  observable  in  the  last  two  stimuli. 
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Figure  5  -  Data  traces  for  subject  F12, 17  seconds  starting  at  time  274  seconds 


Figure  6,  sampled  between  3186  and  3203  seconds  following  initiation  of  the  experiment, 
demonstrates  examples  of  what  we  consider  to  be  “overflow”  movements.  Note  responses  on  all 
three  pressure  sensors.  Following  the  first  stimulus  shown  (central  location),  there  is  a  concurrent 
response  with  mouse  and  pressure  sensor  and  a  concurrent  release  of  pressure  on  the  right  sensor. 
To  the  second  stimulus  (on  the  left),  we  see  concurrent  release  of  both  the  left  and  right  pressure 
sensor  at  response  termination.  The  same  is  seen  on  the  next  trial  (central).  The  minor  pressure 
sensor  changes  on  the  side  sensor,  with  perhaps  the  exception  of  the  response  to  the  last  stimulus, 
do  not  appear  to  be  time  locked  to  other  responses. 
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Figure  6  -  Data  traces  for  subject  F12, 17  seconds  starting  at  time  3186  seconds 

Figure  7  demonstrates  time  locked  responses  from  the  left  pressure  sensor  and  the  side  pressure 
sensor.  As  the  subject  depresses  the  left  mouse  button,  he  exerts  pressure  on  the  side  of  the 
mouse  as  well.  The  last  stimulus  on  this  figure  demonstrates  an  anticipatory  response  in  both  the 
mouse  and  pressure  channels. 

These  figures  are  presented  to  demonstrate  the  advantages  associated  with  using  pressure  sensing 
as  an  additional  input.  We  have  identified  a  number  of  problems  with  our  current  procedure  for 
sensing  pressure  exerted  on  the  mouse  and  hope  to  be  able  to  deal  with  these  problems  in  the 
phase  II  effort. 
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Figure  7  -  Data  traces  for  subject  M13, 17  seconds  starting  at  time  3232 


5.2  Running  memory  task  performance 

In  the  memory  component  of  the  EPVT  subjects  could: 

a.  Respond  accurately 

b.  Miss  a  response 

c.  Enact  a  false  alarm  response 

Average  reaction  time  was  measured  from  initiation  of  the  simple  reaction  time  response  as  was 
also  true  of  false  alarm  responses.  The  average  reaction  time  for  accurate  responses  was  0.709 
seconds,  while  that  for  false  alarms  was  1.221  seconds.  These  differences  are  significant  at  the 
0.0001  level  (df  1,728,  F=177.35).  Thus,  most  false  alarm  responses  required  more  time  for  their 
enactment,  suggesting  that  there  was  more  uncertainty  about  the  enactment  of  such  responses 
than  was  true  of  correct  responses.  There  was  no  time  on  task  effect  for  either  response  latency 
or  frequency  of  occurrence  for  either  correct  or  false  alarm  responses.  These  effects  mirror  those 
generally  seen  in  performance  changes  associated  with  sleep  deprivation.  The  major  effect 
appears  to  be  on  simple  RT  with  no  or  minimal  effect  on  more  cognitively  loaded  variables. 
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5.3  Camera  based  measures  associated  with  response  latency 

5.3.1  Saccadic  eye  movements  to  target  (tT)  and  return  from  target  (rT) 

Subjects  make  saccadic  eye  movements  to  shift  gaze  to  the  target  location  (tT)  and  return  from 
the  target  location  to  the  central  or  other  location  (rT).  We  were  surprised  by  the  frequency  of 
occurrence  of  saccades  during  the  interstimulus  interval  and  suspect  that  it  is  the  latter  saccades 
that  are  most  sensitive  to  ToT  effects. 

5.3.2  Timing  of  saccadic  eye  movements  with  respect  to  response  enactment 

To  target  (tT)  saccades  generally  occur  in  close  proximity  to  the  manual  simple  reaction  time 
response.  We  plan  to  investigate  if  saccade  onset  and  response  onset  become  more  variable  as  .a 
function  of  ToT.  If  the  tT  saccade  follows  the  manual  response  by  more  than  300-400 
milliseconds,  the  subject  will  not  see  the  numerical  information  since  the  RT  information  is 
presented  for  400  ms  following  response  initiation.  Subjects  change  strategy  for  acquiring  new 
information.  Sometimes  there  is  a  series  of  events  where  there  is  no  rT  gaze  shift.  The  subject 
waits  until  the  next  stimulus  occurs  before  shifting  gaze.  On  33%  of  the  trials,  the  subject  does 
not  have  to  change  gaze  location.  We  also  observed  subjects  whose  gaze  shift  to  stimulus 
initiation  was  in  the  wrong  direction.  Whether  more  of  these  occur  as  a  function  of  ToT  will  need 
to  be  investigated.  We  suspect  that  this  may  also  work  in  situations  where  we  are  not  in  control 
of  stimulus  location.  We  find  that  fixation  duration,  when  gaze  shift  is  in  the  wrong  direction,  is 
too  short  for  the  acquisition  of  information  from  the  fixated  site.  Thus  unusually  short  (as  well  as 
unusually  long)  fixation  pauses  may  index  a  period  of  lowered  alertness. 

Average  saccade  velocity  of  tT  saccades  generally  does  not  change  appreciably  as  a  function  of 
ToT;  rT  saccades  are  more  likely  to  have  occasions  when  they  are  slower  than  normal. 

Interstimulus  saccades  are  more  likely  to  be  of  longer  than  normal  duration.  One  can 
occasionally  observe  slow  eye  movements  (SEM),  though  in  most  subjects  these  only  occur  after 
the  eyes  have  closed.  They  may  thus  not  be  useful  in  the  evaluation  of  alertness.  Others  have 
noted  that  when  SEMs  occur,  it  is  too  late  to  avoid  an  accident. 

The  frequency,  as  well  as  amplitude  and  direction,  of  interstimulus  saccades  deserves 
investigation.  One  class  of  such  events  involves  the  occurrence  of  a  gaze  shift  to  a  new  location, 
a  brief  fixation  pause,  and  a  return  to  the  previous  position.  We  refer  to  these  as  “square  wave 
jerks”,  they  are  similar  to  such  “jerks”  seen  as  subjects  track  a  target  and  either  lag  behind  or  get 
too  far  away  from  the  target  location.  More  such  “jerks”  are  seen  in  patients  suffering  from 
schizophrenia,  and  a  common  interpretation  is  that  they  are  associated  with  brief  periods  of 
interruption  of  attention  to  the  task  of  pursuing  the  target  with  the  eyes. 

We  have  included  two  figures  depicting  some  of  the  measures  of  interest.  Figure  8  depicts  both 
horizontal  eye  movements  and  minor  head  movements  (inferred  from  measuring  changes  in 
position  of  the  center  of  the  eyeball).  The  head  movements  of  interest  are  small.  The  first 
stimulus  was  presented  to  the  right,  and  we  see  an  eye  movement  to  the  right  (upward  movement 
in  channel  1).  The  return  movement  is  lost  because  it  coincides  with  camera  data  loss  occasioned 
by  an  eye  blink.  This  figure  thus  nicely  demonstrates  that  blinks  are  unlikely  to  occur  as  gaze 
shifts  to  a  target  location  (tT  saccade)  and  are  most  likely  to  occur  concurrent  with  gaze  returning 
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to  a  central  location  in  anticipation  of  the  next  stimulus.  Notice  also  that  there  is  a  blink  at 
approximately  the  same  latency  from  stimulus  onset  for  the  third  stimulus  presented  at  the 
central  location.  No  eye  movement  was  required  to  detect  this  event.  Notice  also  that  the  second, 
third  and  fourth  stimuli  end  in  odd  integers  requiring  a  response  to  the  running  memory  task.  The 
subject  successfully  makes  this  response,  blinks  during  the  response  with  a  second  blink  shortly 
thereafter. 

Figure  9  depicts  a  very  different  pattern  of  eye  movements  than  that  seen  in  the  previous  figure. 
This  person  is  an  active  explorer  of  his  environment.  The  first  stimulus  is  at  the  central  location, 
and  we  see  an  eye  gaze  shift  to  the  left,  rapid  return  to  center  before  response  enactment.  The 
next  stimulus  is  to  the  right,  and  the  subject  shifts  gaze  to  that  location  from  the  center.  The  blink 
is  not  concurrent  with  an  eye  movement  and  the  eye  movement  returning  gaze  to  the  central 
location  appears  to  be  slower  than  the  move  to  the  target  location.  Again  we  see  a  gaze  shift  to 
the  right  preceding  stimulus  presentation.  The  fourth  stimulus  presentation  occurs  during  a 
period  of  data  loss,  most  likely  a  blink,  and  is  followed  by  a  long  latency  response. 
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Figure  9  -  Data  traces  for  subject  M13, 17  seconds  starting  at  time-2573  seconds 

5.3.3  Head  movements  associated  with  saccades 

These  head  movements  are  inferred  from  changes  in  the  calculated  position  of  the  center  of  the 
eyeball.  The  only  way  it  can  change  is  if  there  is  a  head  movement.  The  observed  movements  are. 
never  greater  in  amplitude  than  5%  of  the  gaze  shift,  and  in  most  instances  do  not  exceed  2%. 
We  have  assured  ourselves  that  these  movements  are  not  artifacts  of  the  software  used  to  identify 
gaze  location.  We  did  this  by  concurrent  collection  of  gaze  information  and  head  movements 
identified  using  the  LDV.  We  attached  a  small  “flag”  to  the  center  of  the  forehead,  aimed  the 
laser  at  the  flag  and  monitored  head  position  in  the  horizontal  plane.  There  was  a  high  level  of 
coincidence  of  the  LDV  and  camera-based  head  movement  measure.  This  reassured  us  that  we 
were  dealing  with  minor  head  movements  and  not  noise. 

Such  head  movements  are  seen  in  conjunction  with  tT  saccades.  Initiation  of  the  head  movement 
component  is  generally  tightly  coupled  to  saccade  onset  with  termination  usually  later  than 
saccade  termination.  Though  we  had  expected  a  dissociation  of  the  initiation  time  of  these  two 
measures  as  a  function  of  ToT,  our  preliminary  analyses  do  not  find  this  to  be  the  case.  We 
propose  further  analyses  to  document  this  observation. 

We  have  not  systematically  evaluated  head  movements  associated  with  other  than  tT  saccades. 
Though  not  dramatic,  one  can  see  head  movements  associated  with  gaze  shift  in  Figure  8.  The 
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first  stimulus  (to  the  right)  has  a  minor  head  movement  concurrent  with  the  gaze  shift  to  the 
right.  A  similar  minor  head  movement  can  be  seen  associated  with  the  gaze  shift  to  the  right  for 
the  next  stimulus. 

5.3.4  Pupil  diameter  measures  reflecting  performance  impairment. 

Pupil  diameter  will  be  a  most  important  measure  in  the  identification  of  attentional  lapses. 

Two  types  of  analyses  were  performed  on  the  pupillary  measure,  one  dealing  with  ToT  effects 
(tonic  activity  measure);  the  other  dealing  with  the  issue  of  differences  in  pupillary  activity 
associated  with  lapses  in  performance  as  indexed  by  long  latency  SRT  responses  (phasic  activity 
measure). 

5.3.4.1  Tonic  activity  measure 

There  is  suggestive  data  in  the  literature  that  fatigued  individuals  have  both  smaller  pupils 
(Loewenstein  and  Loewefeld  1964)  as  well  as  greater  variability  in  pupil  diameter  (Yoss  et  al. 
1970).  The  greater  variability  in  pupil  diameter  has  been  labeled  “pupillary  hippus”.  It  is 
observed  when  subjects  are  required  to  sit  in  a  darkened  room  for  a  30-minute  period  and  look  at 
a  dimly  lit  spot  in  front  of  them.  We  wondered  whether  this  phenomenon  was  situation  specific 
or  if  it  could  be  observed  under  conditions  of  task  performance.  For  the  present  analysis,  we 
sampled  the  pupil  every  17  seconds  and  averaged  those  measures  in  5  minute  increments.  If  no 
data  was  available  at  the  midpoint  of  the  17-second  period  because  of  blinks  or  other  reasons  for 
data  loss,  we  sampled  the  data  shortly  after  it  again  became  available.  Where  data  was  lost  for  a 
17  second  period  we  reduced  the  number  of  samples  in  the  data  set. 

The  information  abstracted  was  average  pupil  diameter  per  5-minute  segment  and  variability  as 
measured  by  S.D.  in  each  five-minute  segment.  The  following  figures,  Figure  10,  Figure  11,  and 
Figure  12,  demonstrate  the  results  for  three  subjects.  Subject  M22  demonstrates  pupillary 
diameter  decreases  over  the  60-minute  period  of  task  performance,  as  well  as  increases  in 
variability  as  reflected  in  the  S.D.  As  pupil  diameter  decreased,  S.D.  increased.  Subject  M15  also 
demonstrated  pupil  diameter  decreases  over  time,  though  not  as  dramatic  as  the  changes  seen  for 
M22.  On  the  other  hand,  variability  was  every  bit  as  dramatic  for  this  subject  as  M22.  Subject 
M14  demonstrates  a  pattern  quite  different  from  the  other  two  in  that  he  demonstrates  an 
increase  in  pupil  diameter  as  a  function  of  ToT.  Note,  however  that  the  measure  of  variability 
demonstrates  an  increase  as  a  function  of  ToT.  We  will,  in  the  near  future  complete  the  analysis 
of  data  for  all  20  subjects.  Should  the  remaining  subjects  demonstrate  a  pattern  of  pupil  diameter 
change  similar  to  that  demonstrated  by  all  three  of  the  presented  subjects,  we  will  then  go  ahead 
and  conduct  spectral  analysis  of  the  data  using  a  procedure  similar  to  that  used  by  McLaren  et  al. 
(1992). 
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5.3.4.2  Phasic  activity  measure 

We  assumed  that  when  subjects  are  responding  slowly  to  the  SRT  task  they  are  at  a  lower  level 
of  alertness  than  when  they  are  responding  “normally”.  Taking  this  one  step  further  allows  us  to 
suggest  that  when  subjects  are  responding  rapidly,  the  pupil  should  be  larger  than  when  they  are 
responding  slowly,  and  hopefully  larger  than  when  they  are  responding  “normally”.  In  addition, 
we  suggested  that  the  enactment  of  a  response  might  be  alerting  to  subjects  who  were  at  a  low 
level  of  alertness  at  time  of  stimulus  presentation.  This  should  be  associated  with  an  increase  in 
pupil  diameter  for  long  latency  responses.  Pushing  this  a  bit  further,  we  would  like  to  suggest 
that  when  subjects  are  reasonably  alert  at  time  of  stimulus  presentation,  there  should  be  no 
change  or  perhaps  a  decrease  in  pupil  diameter  following  response  initiation.  They  know  that  the 
next  stimulus  will  not  come  immediately  following  their  response  and  thus  can  relax  briefly 
before  the  next  stimulus  occurs.  To  evaluate  these  “speculations”  we  performed  the  following 
analyses. 

We  identified  the  50  slowest  and  50  fastest  responses  for  each  subject  (out  of  approximately 
1200  SRT  responses)  and  sampled  50  events  close  in  time  to  the  slowest  responses  and  where  the 
stimulus  was  at  the  same  location  as  the  long  latency  response.  In  addition  to  sampling  pupillary 
data  at  stimulus  presentation,  we  also  sampled  it  at  response  onset,  as  well  as  at  340  milliseconds 
and  1020  milliseconds  following  response  onset.  Where  no  data  was  available  because  of  lid 
closure  or  other  reasons,  we  sampled  the  data  close  in  time  to  the  event  in  question.  If  data  loss 
was  for  an  extended  period,  i.e.  included  stimulus  and  response  onset,  that  data  was  deleted. 
When  we  sampled  at  a  point  in  time  other  than  the  initiation  of  stimulus  and  response  onset  and 
the  two  points  beyond  response  onset  that  data  was  flagged.  We  have  completed  the  analysis  for 
all  20  subjects  used  as  the  data  base  for  our  evaluations.  Figure  13  depicts  the  results  of  these 
analyses.  Repeated  measure  analyses  of  variance  of  the  three  data  points  associated  with  stimulus 
onset,  short,  normal,  and  long  latency  responses  was  significant  at  beyond  the  0.000  level  and 
paired  comparisons  identified  all  paired  comparisons  as  being  significant. 


•Short  Latency  EPVT 
■Normal  Latency  EPVT  | 
■Long  Latency  EPVT 


Time  Point 


Figure  13  -  Pupil  diameter  values  near  stimulus  presentation  for  short,  normal,  and  long  latency  responses 
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Similar  analyses  were  conducted  at  response  onset  as  well  as  the  two  points  beyond  response 
onset.  The  analyses  for  response  onset  and  340  milliseconds  post  response  produced  the  same 
results  as  the  analysis  to  stimulus  onset.  The  analysis  of  data  sampled  1020  milliseconds  after 
response  onset  finds  no  difference  between  the  long  and  normal  latency  response  data  while  both 
are  significantly  different  from  the  short  latency  response  data. 

What  does  the  individual  pupillary  response  look  like?  We  present  data  from  one  subject  who 
provides  us  with  many  events  where  a  long  latency  response  is  associated  with  the  pupil  being 
small  at  time  of  stimulus  presentation.  Not  all  subjects  demonstrate  as  nice  a  pattern  as  seen  in 
this  subject. 
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Figure  14  -  Data  traces  for  subject  F10, 17  seconds  starting  at  time  2014  seconds 

In  Figure  14,  the  third  stimulus  (right),  a  stimulus  with  a  long  latency  response,  has  pupil 
diameter  of  3.238  mm  at  the  point  immediately  preceding  a  blink.  This  blink  follows  stimulus 
onset  and  occurs  before  response  onset.  For  comparison,  we  have  identified  pupil  diameter  at  a 
point  prior  to  the  initiation  of  the  pupil  constriction  response.  Pupil  diameter  at  this  point  is  4.496 
mm.  The  decrease  in  pupil  diameter  between  these  two  points  is  1.258  mm. 

In  Figure  15,  following  the  response  to  the  first  stimulus  (center),  we  have  a  series  of  eyelid 
closures  as  well  as  blinks.  Pupil  diameter  at  the  point  of  response  initiation  for  the  second 
stimulus  (right),  a  stimulus  with  a  long  latency  response,  is  3.314  mm.  The  two  other  measures 
of  pupil  diameter  are  provided  to  give  some  idea  of  the  pupil  constriction  response  preceding 
long  latency  responses.  Note  that  following  this  long  latency  response  the  subject  goes  back  to 
demonstrating  normal  duration  blinks  as  well  as  normal  latency  responses. 
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Figure  15  -  Data  traces  for  subject  F10, 17  seconds  starting  at  time  2214  seconds 


5.4  Cardiovascular  measures 

5.4.1  Cardiovascular  measures  associated  with  “expectancy” 

Description  of  analyses  performed 

This  analysis  dealt  with  the  evaluation  of  cardiac  activity  during  periods  of  “expectancy”  or 
anticipation.  Prior  research  from  our  as  well  as  other  laboratories  has  demonstrated  that  heart 
period  duration  increases  when  subjects  are  anticipating  a  relatively  infrequently  occurring  event 
(Sirevaag,  et  al.  1999).  We  sought  to  confirm  this  in  the  preliminary  data  collected  with  the 
EPVT. 

Data  was  evaluated  for  five  subjects.  For  four  of  the  subjects,  the  analysis  was  conducted  on  data 
collected  with  the  LDV,  and  we  used  the  opening  of  the  aortic  valve  to  trigger  identification  of 
the  inter-beat  interval.  For  the  fifth  subject  standard  ECG  data  was  available  and  the  peak  of  the 
R  wave  was  used  to  identify  EBI’s.  Data  analysis  utilized  a  computer  routine  for  identifying  the 
peak  of  the  R  wave  and  was  manually  edited  for  artifacts.  The  major  purpose  of  this  analysis  was 
to  determine  whether  we  could  reliably  collect  cardiovascular  data  with  the  LDV  under  the 
conditions  of  this  experiment.  The  answer  to  the  latter  question  is  YES. 

The  first  analysis  of  heart  period  change  associated  with  anticipation  involved  measuring  the 
duration  of  the  five  heart  periods  preceding  and  following  occurrence  of  a  third  odd  integer  (O- 
O-O-R).  Since  anticipation  was  not  limited  to  the  occurrence  of  three  odd  integers,  but 
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anticipation  that  the  integer  following  a  series  of  two  odd  integers  would  be  odd,  a  further 
analysis  was  conducted  on  the  five  beats  preceding  and  following  the  memory  task  response 
where  no  response  was  required,  i.e.  high  level  of  expectation  and  disconfirmation  of  the 
expectation  (O-O-E-  no  response). 

It  was  expected  that  anticipation  of  the  third  odd  integer  would  lead  to  lengthening  of  the  heart 
period.  In  our  earlier  study  the  inter-stimulus  interval  was  constant,  subjects  could  thus  not  only 
anticipate  the  occurrence  of  an  event  requiring  a  response  but  also  could  anticipate  time  of 
occurrence.  In  the  present  study  the  inter-stimulus  interval  was  varied  so  that  anticipation  of 
when  a  stimulus  occurred  was  not  available. 

We  evaluated  the  inter-beat  interval  (IB I)  for  the  five  periods  preceding  onset  of  the  third  integer, 
the  period  involving  presentation  of  the  third  integer  and  the  five  periods  following  the  third 
integer.  (Since  the  inter-stimulus  interval  varied  around  an  average  of  2.5  "s,  this  analysis 
frequently  included  periods  where  stimuli  were  presented.)  Three  analyses  were  conducted,  one 
for  the  situation  where  a  response  was  required  and  enacted  (with  response),  one  for  events 
where  a  response  was  required  but  not  made  (missed  signal  -  no  response)  and  for  the  situation 
where  two  odd  integers  were  followed  by  an  even  integer  (3rd  even).  The  analysis  was  conducted 
both  for  “raw”  data  as  well  as  for  “normalized”  data.  Data  was  normalized  using,  for  each 
subject,  the  average  of  the  first  IBI  as  the  “comparison”  value  and  expressing  successive  average 
values  for  each  individual  as  a  percentage  of  the  first  interval.  An  analysis  of  variance  was  done 
on  heart  period  duration  following  normalization.  The  results  demonstrated  a  significant 
difference  across  the  inter-beat  intervals  with  a  p<  0.01. 

Figure  16  depicts  the  results  of  the  heart  period  analysis  both  for  average  (i.e.  “raw”  heart 
periods)  as  well  as  normalized  values  under  the  condition  in  which  a  response  to  the  memory 
task  was  required. 

As  depicted  in  both  Figure  16  and  Figure  17,  the  heart  period  for  the  interval  involving 
presentation  of  the  third  integer  and  for  the  interval  following  it  demonstrate  a  lengthening  of 
heart  period  for  both  the  conditions  where  a  response  was  required  and  made,  and  the  condition 
where  anticipation  of  the  third  odd  integer  was  discontinued  and  no  response  was  made.  The 
analysis  of  variance  showed  a  significant  difference  between  the  heart  periods  with  a  p-value  of 
<0.01. 

In  the  “with  response”  condition  we  evaluated,  for  each  of  the  20  trials  analyzed  for  each  subject, 
the  period  in  which  the  longest  inter-beat  interval  occurred.  For  two  subjects,  the  heart  period 
preceding  the  response  was  most  often  the  longest.  In  the  other  three  subjects  this  heart  period 
contained  the  longest  period  the  second  most  times. 

The  next  analysis  evaluated  the  condition  where  two  odd  integers  were  followed  by  an  even 
integer.  Approximately  30  occurrences  of  this  sequence  were  analyzed  for  each  subject. 

An  analysis  across  all  subjects  demonstrated  cardiac  deceleration  in  the  interval  containing  the 
third  (even)  integer,  and  the  period  directly  following  it.  The  pattern  was  similar  to  that  seen  in 
the  analysis  above.  An  analysis  of  variance  done  on  the  data  from  all  subjects  after  normalization 
showed  a  significant  difference  with  p<  0.01. 
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Figure  17:  Raw  and  normalized  heart  period  for  2 
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The  last  analysis  was  conducted  to  determine  if  heart  period  slowing  occurs  when  a  subject 
should  be  showing  anticipation  but  does  not.  These  instances  occur  when  a  sequence  of  three  odd 
integers  is  not  followed  by  a  response.  This  event  occurred  relatively  infrequently  and  the 
average  number  of  such  events  was  5.4  times  per  subject.  This  analysis  was  done  with  the  five 
heart  periods  before  the  third  odd  integer  and  the  seven  periods  after  the  integer.  No  significant 
effects  were  obtained.  Figure  18  suggests  a  decrease  in  heart  period  both  preceding  and 
following  presentation  of  the  third  integer,  but  these  effects  were  not  reliable. 

After  all  three  conditions  were  analyzed,  a  further  analysis  was  done  comparing  the  three 
conditions  (O-O-O-R;  O-O-O-no  response;  and  O-O-E-no  response).  An  analysis  of  variance 
was  done  across  all  three  groups  to  determine  if  there  was  a  significant  difference  in  the  pattern. 
The  results  showed  a  difference  with  a  p<  0.01  in  the  normalized  data.  Subsequently,  t-Tests 
were  conducted.  They  demonstrated  no  differences  between  the  correct  response  condition  and 
the  condition  with  the  third  integer  even  (p>.2)  for  any  of  the  comparisons.  The  t-Tests 
demonstrated  that  the  condition  with  three  odd  integers  and  no  response  was  significantly 
different  from  the  other  two  conditions  (p<0.05). 

Figure  19  shows  both  the  raw  average  heart  period  and  the  normalized  heart  period  data  for  all 
three  conditions. 


31 


Bio-Behavior 


Analysis  Systems 


Contractor: 
Contract  No.: 
Project  Title: 
Report  Title: 


Date: 


Bio-Behavior  Analysis  Systems,  LLC 

FA9550-04-C-0082 

Real-Time  Detector  of  Human  Fatigue 

Final  Technical  Report:  Detection  of  Human 

Fatigue 

05  Aug  2005 


Bio-Behavior 
Analysis  Systems 


Contractor:  Bio-Behavior  Analysis  Systems,  LLC 
Contract  No.:  FA9550-04-C-0082 
Project  Title:  Real-Time  Detector  of  Human  Fatigue 
Report  Title:  Final  Technical  Report:  Detection  of  Human 
Fatigue 

Date:  05  Aug  2005 


Figure  19:  Raw  and  normalized  heart  period  for  all  three  conditions  described 

Discussion 

Previous  research  (Lacey,  1974;  Jennings,  1992;  van  der  Molen,  2000;  Somsen,  2004)  using 
principally  fixed  foreperiod  reaction  time  tasks  demonstrated  that  in  response  to  visual  stimuli 
and  anticipation  the  heart  rate  of  a  subject  will  slow.  The  studies  show  that  the  anticipation 
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response  is  greatest  when  stimulus  onset  is  predictable.  It  is  attenuated  in  conditions  where 
timing  of  stimulus  presentation  is  not  predictable,  as  was  the  case  in  our  study.  Our  data  support 
the  contention  that  the  effect  is  still  present  when  stimulus  timing  is  not  predictable.  We  show 
that  when  subjects  are  correctly  anticipating  (the  correct  response  condition,  and  the  no  third  odd 
integer  condition)  heart  rate  slows  in  response  to  the  third  integer.  In  the  condition  where 
subjects  should  be  responding  but  fail  to  respond,  such  cardiac  slowing  is  not  present.  Our  results 
thus  demonstrate  that  it  is  the  anticipation  of  a  stimulus  requiring  a  response  and  not  the 
enactment  of  the  response  that  is  responsible  for  the  cardiac  deceleration.  When  anticipation  is 
not  present,  i.e.,  the  condition  where  a  response  is  required  but  not  made,  the  cardiac  slowing 
does  not  materialize. 

This  slowing  is  presumed  to  be  due  to  vagal  stimulation  of  the  heart,  this  supposition  has  in  turn 
been  supported  by  empirical  physiological  evidence  (Somsen;  2004).  Previous  studies  by  Lacey 
(1974,  1977,  1978)  and  a  review  of  this  and  other  studies  by  Somsen  (2004)  demonstrate  that  a 
visual  stimulus  can  affect  the  inter-beat  interval  (IBI)  if  it  is  presented  early  enough  in  the 
interval,  or  the  interval  directly  following  it  if  it  is  presented  in  the  later  part  of  the  EBI.  In  the 
present  study,  no  attempt  was  made  to  have  stimulus  presentation  coincide  with  a  particular 
phase  of  the  cardiac  cycle.  The  wider  distribution  of  heart  period  increase  found  in  associated 
with  anticipation  may  reflect  the  fact  that  the  imperative  stimulus  was  presented  at  random 
phases  in  the  cardiac  cycle. 

Since  our  LDV  technology  in  combination  with  EKG  will  allow  us  to  fractionate  the  cardiac 
cycle  into  three  components  (pre-ejection  period;  ejection  period;  recovery  period)  we  will  be 
able  to  determine  whether  all  components  of  the  cardiac  cycle  are  equally  affected  by 
“anticipation”  or,  as  we  suspect,  the  effect  will  discriminate  between  the  three  components. 

5.4.2  Fractionation  of  heart  period  into;  pre-ejection;  left  ventricular 
ejection,  and  post  ejection  periods. 

There  is  no  research  literature  dealing  with  these  measures  since  they  can  normally  only  be 
obtained  by  invasive  procedures.  We  wondered  whether  there  would  be  changes  in  the 
proportion  of  time  occupied  by  these  components  as  the  interbeat  interval  increased.  This 
analysis,  at  the  present  time,  is  partially  computerized  and  partially  under  operator  control.  Thus 
only  relatively  few  comparisons  have  been  made. 

The  first  investigation  asked  whether  heart  period  slowing  associated  with  periods  of  non¬ 
responding  to  the  task  had  a  similar  distribution  of  these  components  as  heart  period  slowing 
associated  with  expectancy  (see  section  5.4.1  above).  For  a  limited  data  sample,  we  were  not 
able  to  identify  major  differences  in  these  components  across  the  two  conditions.  Further 
analyses  will  be  conducted. 

We  are  also  evaluating  heart  period  variability  as  a  function  of  ToT.  This  evaluation  is  based,  in 
part,  on  our  finding  of  pupillary  hippus  increasing  as  a  function  of  ToT.  We  are  sampling  the 
ECG  for  successive  5-minute  periods  and  calculating  both  mean  and  standard  deviation  of  the 
interbeat  interval.  For  the  few  subjects  evaluated  to  date,  there  appears  to  be  an  increase  in 
variability  as  a  function  of  ToT.  This  might  then  become  an  additional  measure  for  the 
evaluation  of  tonic  effects  indicative  of  fatigue. 
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What  does  the  LDV  cardiac  signal  look  like?  Figure  20  provides  an  example.  The  first  channel  is 
a  standard  ECG;  the  second  is  the  concurrently  recorded  LDV  signal  from  the  carotid  artery.  The 
first  major  peak  in  the  LDV  signal  identifies  onset  of  the  opening  of  the  aortic  valve,  the  next 
major  peak,  with  a  dicrotic  notch,  identifies  the  closing  of  the  valve.  In  combination  with  the 
ECG,  we  can  thus  identify  the  pre-ejection  period  (from  peak  of  ECG  r-wave  to  the  first  peak  in 
the  LDV  signal;  the  left  ventricular  ejection  period  (from  onset  of  the  first  LDV  peak  to  the  onset 
of  the  dicrotic  notch)  and  a  final  “rest”  period.  Displayed  are  4  seconds  of  data. 


LDV  Carotid  Artery 


Mouse  Right 
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Stimulus  Left 


Stimulus  Right 


Figure  20  -  Data  traces  for  subject  M13, 2  seconds  starting  at  952  seconds 
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6  Conclusions 

The  EPVT  task  developed  for  this  project  appears  to  be  sensitive  to  time  on  task  effects  even 
with  well-rested  subjects.  We  have  demonstrated  the  capability  for  recording  oculomotor  as  well 
as  Laser-Doppler  Vibrometry  data  under  these  conditions.  We  further  have  demonstrated 
significant  time  on  task  effects  for  the  bio-behavioral  measures  employed.  Analysis  of  data  from 
some  biological  systems  was  completed  for  the  twenty  subjects.  We  were  able  to  demonstrate 
that  pupil  diameter  was  an  excellent  reflector  of  lapses  in  alertness.  When  subjects  were  at  a 
lower  level  of  alertness,  as  reflected  by  long  latency  reaction  time  responses,  pupil  diameter  was 
significantly  smaller  at  time  of  stimulus  onset,  as  well  as  at  response  onset,  than  when  response 
latencies  were  “normal”  or  rapid.  Heart  period  data  demonstrated  that  anticipation  of  responding 
to  the  running  memory  task  produced  significant  slowing  of  heart  rate.  This  is  concordant  with 
the  literature.  We  are  well  along  in  specifying  both  hardware  and  software  requirements  to  allow 
for  the  on-line  analysis  of  bio-behavioral  signals  of  interest. 
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ABSTRACT 

A  hybrid  sustained  attention  task  was  developed  in  order  to  examine  the  relationships  between  manual  response  times 
and  the  timing  and  morphology  of  horizontal  saccades  involved  in  shifting  gaze  to  a  source  of  task  relevant  visual 
information.  Twelve  subjects  performed  this  task  for  60  min  with  no  breaks.  Performance  and  gaze  control  measures 
were  aggregated  across  20  min  intervals  comprising  early,  middle  and  late  segments  of  the  task.  Response  time 
variability  was  significantly  increased  during  later  task  segments  (p<0.05).  These  segments  were  also  associated  with 
increased  variability  in  the  amplitude  of  saccades  (p<0.05).  Saccade  durations  during  the  late  task  segments  were  also 
longer  and  more  variable  (p<0.05).  Correlations  between  response  times  and  measure  of  saccadic  activity  were  also 
computed  across  consecutive  5  min  intervals  for  each  individual  subject.  The  obtained  correlations  between  saccade 
latency  and  response  times  exceeded  0.70  for  six  of  the  twelve  subjects.  Additional  analyses  examined  the  relationship 
between  trials  characterized  by  extreme  values  on  either  the  performance  or  the  gaze  control  measures.  Trials 
characterized  by  extremely  long  response  times  were  also  associated  with  increased  saccade  amplitudes,  durations  and 
latencies  (p<0.01).  Conversely,  response  times  were  abnormally  long  on  trials  categorized  as  extreme  on  the  basis  of 
the  saccade  morphology  and  timing  measures  (p<0.01).  These  results  confirm  the  utility  of  the  sustained  attention  task 
as  a  laboratory  platform  for  the  development  of  real-time  systems  for  alertness  monitoring.  The  data  also  support  the 
contention  that  measures  of  gaze  control  behavior  can  reflect  aspects  of  cognitive  activity  and,  therefore,  should  be 
seriously  considered  for  inclusion  in  any  physiologically-based  alertness  assessment  battery. 

Keywords:  gaze  control,  saccades,  oculomotor,  vigilance,  fatigue,  cognition 

1.  INTRODUCTION 

Many  tasks  in  a  variety  of  operational  settings  require  individuals  to  monitor  aspects  of  the  environment  and 
infrequently  intervene  in  the  ongoing  operation  of  a  particular  system.  An  operator’s  ability  to  maintain  a  high  level  of 
performance  under  such  “vigilance”  conditions  is  limited.  Deficits  in  performance  increase  as  a  function  of  Time-On- 
Task  (TOT)  and  typically  become  manifest  within  the  first  hour  of  task  performance.  However,  the  decline  in 
performance  is  typically  not  a  linear  function.  Instead,  increases  in  the  frequency  and/or  duration  of  behavioral  lapses, 
or  blocks,  surrounded  by  periods  of  relatively  normal  performance  are  typically  found  with  increased  TOT1. 

A  key  issue  is  whether  these  performance  lapses  are  reliably  associated  with  or  preceded  by  physiological  signs 
predictive  of  impending  performance  impairments.  Our  research  program  has  been  designed  to  address  three  key 
issues.  First,  can  laboratory  tasks  be  used  to  identify  measures  associated  with  performance  blocks?  Second,  can  the 
results  obtained  in  the  laboratory  be  extrapolated  to  real-world  environments?  Finally,  can  these  measures  be  obtained 
and  evaluated  in  real-time  to  provide  instantaneous  evaluations  of  operator  alertness  and/or  cognitive  load?  The 
ultimate  goal  of  this  program  of  research  is  the  development  and  validation  of  a  real-time  alertness  monitoring  system. 
Furthermore,  in  order  to  assure  the  widest  possible  applicability  in  real-world  environments,  the  proposed  system  will 
rely  upon  non-contact  sensors  for  monitoring  physiological  processes  and  will  not  impose  additional  demands  (i.e. 
“secondary  tasks”  unrelated  to  the  normal  operational  environment)  upon  the  operator.  Such  a  system  would  have  a 
wide  range  of  applications  in  areas  as  diverse  as  fitness  for  duty  assessment;  adjustment  of  the  optimal  level  of 
automation  in  man-machine  systems;  titration  of  task  difficulty  during  computer-aided  learning;  the  evaluation  of 
operator  alertness/efficiency  during  sustained  performance  and/or  following  sleep  deprivation,  etc. 

Measures  of  oculomotor  activity  are  attractive  candidates  for  inclusion  in  a  real-time  alertness  assessment  battery.  The 
gaze  control  system  is  intimately  involved  in  the  acquisition  and  processing  of  information2.  The  direction  of  gaze,  the 
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duration  for  which  gaze  is  directed  at  a  given  location,  and  the  speed  with  which  the  eyes  move  to  a  target  location  all 
reflect  aspects  of  information  processing  and  attention.  Movements  of  the  head  also  participate  in  the  acquisition  of 
visual  information  and  can,  therefore,  potentially  provide  additional  insights  into  the  processing  demands  associated 
with  task  performance.  The  behavior  of  the  eyelids  is  also  sensitive  to  aspects  of  information  processing.  The 
frequency,  duration  as  well  as  the  timing  of  blinks  with  respect  to  external  events  can  provide  a  great  deal  of 
information  about  the  current  state  of  an  operator.  Pupil  diameter  change  as  a  reflector  of  variables  such  as  workload 
and  other  cognitive  operations  can  also  be  obtained  with  most  camera-based  systems.  Importantly,  the  advent  of 
camera-based  eye  trackers  means  that  many  aspects  related  to  gaze  control  can  be  measured  in  real-time  without  the 
application  of  electrodes  or  other  obtrusive  transducers. 

Measures  of  gaze  control  can  be  further  divided  as  a  function  of  whether  the  oculomotor  activity  in  question  is 
specifically  tied  to  elements  of  the  task  being  performed  by  the  operator.  For  example,  the  PERCLOS  measure  of  lid 
closure3  and  other  metrics  such  as  blink  frequency  do  not  depend  on  the  architecture  of  the  specific  task  being 
performed  by  the  operator.  In  contrast,  measures  of  blink  and  saccade  inhibition  as  a  consequence  of  the  anticipation  of 
the  delivery  of  task-relevant  information  cannot  be  obtained  in  the  absence  of  knowledge  concerning  the  dynamics  of 
the  task  being  performed.  The  latency  of  saccadic  eye  movements  directing  gaze  to  the  location  of  a  source  of  mission 
critical  information  is  another  example  of  a  task-specific  measure  of  gaze  control.  We  believe  that  both  general  and 
task-specific  aspects  of  gaze  control  behavior  will  be  important  components  of  our  real-time  assessment  battery.  In  this 
report,  we  focus  upon  attempts  to  relate  performance  on  a  sustained  attention  task  to  aspects  of  a  task-specific 
component  of  gaze  control  --  namely  To  Target  (TT)  horizontal  saccades  which  shift  gaze  to  a  source  of  task  relevant 
visual  information. 

To  this  end,  a  hybrid  sustained  attention  task  was  developed.  Our  Enhanced  Psychomotor  Vigilance  Task  (EPVT) 
consisted  of  a  simple  visual  Reaction  Time  (RT)  task,  similar  to  traditional  PVT  paradigms4,  differing  by  the  addition  of 
spatial  and  memory  task  components.  Subjects  performed  this  task  continuously  for  60  min.  Relationships  between  RT 
and  TT  saccade  timing  and  morphology  were  evaluated  across  consecutive  20  min  intervals  and  also  across  consecutive 
5  min  intervals.  Additional  analyses  focused  upon  groups  of  trials  identified  as  extreme  (very  fast  vs.  very  slow  RTs, 
for  example)  were  also  conducted. 


2.  METHODOLOGY 


2.1  Subjects 

Twelve  individuals  (8  male,  4  female)  ranging  in  age  between  18  and  25  years  participated  in  the  study.  All  subjects 
were  in  good  health  and  possessed  normal  or  corrected-to-normal  vision  (no  contact  lenses)  and  hearing.  The  subjects 
were  screened  for  previous  history  of  head  trauma  and  neurological  disorders. 

2.2  Tasks  and  stimuli 

The  EPVT  required  subjects  to  respond  as  quickly  as  possible  (by  pressing  the  left  button  on  a  mouse)  following  the 
activation  of  a  timer  presented  either  in  the  center  of  the  screen  or  offset  10  degrees  to  the  left  or  right  of  the  midpoint. 
The  manual  response  terminated  the  timer  and  the  resulting  Reaction  Time  (RT)  information  remained  displayed  for 
400  msec.  The  interval  between  the  disappearance  of  the  RT  feedback  and  the  presentation  of  the  next  stimulus  varied 
between  1.5  and  3.5  sec  (mean  interval  =  2.5  sec).  If  no  response  occurred  within  4000  msec  the  trial  was  labeled  a 
miss.  A  second  response  (depression  of  the  right  mouse  button)  was  required  following  three  successive  reaction  times 
ending  in  an  odd  integer.  The  value  of  the  last  digit  of  the  displayed  RT  was  under  experimenter  control  and  was 
manipulated  so  that  48  events  involving  the  occurrence  of  three  odd  integers  and  96  instances  of  two  odd  integers 
followed  by  an  even  integer  were  equally  distributed  across  the  60  min  duration  of  the  task.  This  eye-movement  rich 
task,  therefore,  embodied  both  simple  RT  (frequent  responses  to  timer  presentations)  and  memory  task  (infrequent 
responses  to  three  consecutive  odd  RTs)  components. 

2.3  Apparatus 

Oculomotor  behaviors  (horizontal  and  vertical  movements,  pupil  diameter  and  head  movements)  were  monitored  using 
a  camera-based  eye  tracking  system  (EyeGaze  -  L.C.  Technologies  Inc.,  Fairfax  VA).  Analog  signals  from  the  eye¬ 
tracking  system  and  event-markers  indicating  the  timing  of  task  stimuli  and  subjects  responses  were  digitized  using  a 
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BIOP  AC  MP100  (Linton  Instrumentation  UK)  workstation.  The  analog  data  were  continuously  sampled  at  2000  Hz 
and  stored  to  disk  using  the  Acknowledge  Software  (version  3.8.1)  associated  with  the  MP100  system. 

2.4  Procedure 

Each  subject  participated  in  a  single  session  which  typically  lasted  2  hours.  Sessions  began  at  10am,  noon,  2pm  or  4pm 
(session  start  times  distributed  equally  across  subjects).  Upon  arrival  in  the  laboratory,  subjects  filled  out  an  informed 
consent  form  as  well  as  a  battery  of  questionnaires  designed  to  assess  their  current  mood,  level  of  alertness,  sleep  habits 
and  the  quality  of  sleep  the  preceding  night.  The  eye-tracker  system  was  then  calibrated  and  the  subjects  were 
presented  with  a  written  set  of  instructions  detailing  the  requirements  of  the  EPVT.  A  five  min  practice  task  was  then 
conducted  to  insure  that  the  subjects  understood  the  instructions  and  that  the  equipment  was  working  properly.  The 
practice  block  was  followed  by  60  min  of  continuous  performance  on  the  EPVT  with  no  breaks  during  this  period. 

2.5  Analyses 

Analyses  were  focused  upon  measures  of  gaze  shifts  which  moved  the  eye  to  the  location  of  the  most  recently  activated, 
peripherally  located  timer.  Software  developed  in  the  authors’  laboratory  [Bio-Behavioral  Data  Reduction  System 
(BBDRS)  version  5.5.4,  Bio-Behavioral  Analysis  Systems,  LLC]  was  utilized  to  identify  and  isolate  TT  saccades  as 
well  as  to  quantify  the  amplitude  and  duration  of  these  saccades  using  standard  algorithms  described  elsewhere5.  In 
addition,  TT  saccade  latency  (interval  between  the  activation  of  a  timer  and  the  initiation  of  an  eye-movement  to  that 
location)  was  also  computed.  The  amplitude  of  minor  head  movements  associated  with  gaze  shifts  was  inferred  from 
the  center-of-the-eyeball  coordinates  computed  by  the  eye-tracking  system.  Simple  RT  was  computed  as  the  latency  of 
the  button  press  response  following  the  activation  of  a  timer. 

Questions  concerning  the  consequences  of  increased  Time-on-Task  (TOT)  upon  these  gaze  control  and  performance 
measures  were  explored  by  segmenting  the  60  min  of  continuous  performance  into  three  20  min  intervals  designated 
“Early”,  “Middle”  and  “Late”.  TOT  effects  were  statistically  evaluated  by  submitting  measures  from  these  time  points 
to  a  single  factor  repeated  measures  Analysis  of  Variance  (ANOVA).  The  degrees  of  freedom  were  adjusted  using 
Geisser  Greenhouse  corrections  for  violations  of  the  sphericity  assumption.  Planned  contrasts  between  the  three  task 
intervals  were  conducted  using  a  modified  Bonferonni  technique6.  Given  the  exploratory  nature  of  this  project,  contrast 
analyses  were  conducted  for  all  significant  main  effects  (p<0.05)  as  well  as  for  non-significant  trends  (p<0.10).  More 
finely-grained  analyses  of  the  relationship  between  specific  gaze  control  measures  and  performance  were  obtained  by 
correlating  the  responses  averaged  across  successive  5  min  intervals  throughout  the  duration  of  the  task.  Due  to  high 
variability  in  both  the  performance  and  oculomotor  measures  in  the  last  5  min  block,  these  correlation  analyses  were 
restricted  to  the  first  55  min  of  task  performance. 


The  relationship  between  the  control  of  gaze  and  performance 
was  also  assessed  through  the  analyses  of  “extreme”  trial 
groups,  which  were  not  explicitly  defined  in  terms  of  TOT. 
Thus,  for  each  subject,  individual  reaction  times  were  sorted  in 
order  to  select  20  trials  in  each  of  three  categories.  “Short  RT’ 
trials  were  defined  as  the  20  fastest  RT  trials.  “Medium  RT’ 
trials  were  identified  by  selecting  20  trials  centered  around  the 
mean  RT  for  the  entire  run.  Finally,  “Long  RT”  trials  were 
defined  as  the  20  longest  RT  trials.  Measures  of  oculomotor 
activity  associated  with  each  of  these  three  response  categories 
were  than  computed  and  submitted  to  statistical  procedures 
identical  to  that  described  for  the  analyses  of  TOT  effects. 
Similar  procedures  were  subsequently  employed  to  examine  the 
extent  to  which  RT  varied  as  a  function  of  associated  gaze 
control  activity.  Thus,  mean  RT  was  computed  across  twenty 
trials  associated  with  “small”,  “medium”  and  “large”  TT 
saccade  amplitudes,  “short”,  “medium”  and  “long”  TT  saccade 
durations,  and  “short”,  “medium”  and  “long”  TT  saccade 
latencies. 
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Figure  1:  RT  by  Task  Segment 
*p<0.05 
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3.  RESULTS  AND  DISCUSSION 


3.1  TOT  effects 

3.1.1  RT  measures 

Performance,  as  indexed  by  the  simple  reaction  time 
measure,  was  not  constant  throughout  all  60  min  of  the  task 
(see  Figure  1).  While  the  TOT  effect  just  failed  to  reach 
statistical  significance  with  respect  to  mean  RT 
[F(2,22)=3.621;  p=0.058],  RT  variability,  reported  as  the 
standard  deviation  of  RT)  was  sensitive  to  TOT 
[F(2,22)=4.746;  p=0.029].  Inspection  of  Figure  1  reveals 
a  trend  towards  longer  RTs  and  increased  response 
variability  with  increasing  TOT  (however  only  the 
differences  between  the  first  and  last  task  segments  could 
be  statistically  discriminated). 

3.1.2  TT  saccade  measures 

Results  of  the  analyses  of  TOT  upon  the  gaze  control 
measures  are  presented  in  Table  1.  Saccade  duration  was 
the  measure  most  sensitive  to  the  TOT  manipulation. 

Saccade  duration  increased  and  became  more  variable  with 
increased  TOT.  Mean  saccade  amplitude  did  not  vary 
across  the  three  task  segments;  however  amplitude 
variability  did  increase  with  increasing  TOT.  The  saccade 
latency  measures  were  not  sensitive  to  TOT.  As  can  be 
seen  in  Figure  2,  saccade  amplitude  variability  increased  significantly  across  all  three  (early,  middle  and  late)  task 

segments.  Mean  saccade  duration  and  saccade  duration  variability 
showed  a  similar  trend  but  only  the  early  and  late  segments  could 
be  discriminated  statistically.  The  saccade  latency  measures 
displayed  a  similar  tendency  to  increase  with  increasing  TOT,  but 
the  only  difference  to  achieve  statistical  significance  was  between 
the  early  and  late  task  segment  on  the  latency  variability  measure. 

3.2  Correlations  between  performance  and  gaze  control 
The  division  of  the  task  into  early,  middle  and  late  segments, 
though  not  uncommon  in  the  literature,  represents  a  fairly  coarse 
and  somewhat  arbitrary  means  of  examining  behavioral  variability. 
To  facilitate  a 
more  finely 
grained 

analysis  of  the  relationship  between  local  task  performance  and 
gaze  control,  mean  RT  and  TT  saccade  characteristics  were 
computed  across  consecutive  five  min  periods.  The  correlations 
between  RT  and  the  gaze  control  measures  were  then  computed 
across  these  intervals  —  separately  for  each  of  the  twelve  subjects. 

The  results  of  this  analysis,  presented  in  Table  2,  clearly  show  that 
aspects  of  gaze  control  are  quite  predictive  of  RT.  TT  saccade 
latency  was  the  best  single  predictor  of  local  RT,  with  obtained 
correlation  coefficients  ranging  between  0.71  and  0.94  for  six  of 
the  twelve  subjects.  This  comparison  yielded  coefficients  lower 
than  0.37  for  only  two  of  the  twelve  subjects  in  the  study. 

Interestingly,  for  both  of  these  subjects,  performance  was 
correlated  with  either  the  TT  saccade  amplitude  or  latency  measure 

at  a  value  at  or  above  0.50.  Table  2:  Single  subject  RT  and  saccade 

correlations 


RT  Correlations  (Pearson’s  R) 


Subjec 

Saccade 

Saccade 

Saccade 

t 

Amplitude 

Duration 

Latency 

1 

-0.25 

0.58 

0.94 

2 

0.88 

0.70 

0.85 

3 

0.16 

0.10 

0.87 

4 

0.20 

0.44 

0.39 

5 

-0.02 

0.50 

0.06 

6 

-0.61 

0.92 

0.71 

7 

-0.15 

-0.05 

0.38 

8 

0.61 

0.39 

0.84 

9 

0.52 

-0.15 

0.18 

10 

0.61 

0.20 

0.45 

11 

0.11 

0.53 

0.93 

12 

-0.38 

0.22 

0.37 

TT  Saccade  Measure 
Amplitude  (mean) 
Amplitude  (std.  dev.) 
Duration  (mean) 
Duration  (std.  dev.) 
Latency  (mean) 
Latency  (std.  dev.) 


F(2,22) 

p- value 

1.029 

0.358 

14.844 

0.001* 

9.763 

0.001* 

5.017 

0.018* 

2.179 

0.137 

2.186 

0.148 

Table  1:  TT  Saccade  by  TOT  ANOVA  results 
*  p  <  0.05 


Task  Segment  Task  Segment 

Figure  2:  TT  saccade  measures  by  Task  Segment.  A) 
Mean  Amplitude;  B)  Amplitude  Variability;  C)  Mean 
Duration;  D)  Duration  Variability;  E)  Mean  Latency; 
F)  Latency  Variability.  *p<0.05 


Appendix  A:  Page  4 


Contract  No:  FA9550-04-C-0082 
Final  Technical  Report  -  Appendix  A 

Related  Paper  Recently  Presented  at  SPIE _ _ _ 

3.3  Extreme  group  comparisons 
3.3.1  Segregation  by  RT 

In  this  analysis,  single  trial  RTs  from  the  entire  task  were  sorted.  Gaze  control  measures  associated  with  the  20  shortest 
and  20  fastest  RTs  were  then  computed.  In  addition,  gaze  control  measures  associated  with  20  medium  latency  RT 


Figure  3:  Saccade  Amplitude  by  RT 
*p<0.05 


Figure  4:  Saccade  Duration  by  RT 

**p<0.01 


Figure  5:  Saccade  Latency  by  RT 

**p<0.01 


trials  (centered  upon  the  mean  of  the  distribution)  were  also  obtained.  TT  saccade  amplitude  [F(2,22)=5.733;  p=0.014] , 
duration  [F(2,22)=16.344;  p<0.001]  and  latency  [F(2,22)=37.407;  p<0.001]  all  varied  significantly  as  a  function  of 
associated  RT.  As  demonstrated  in  Figure  3,  short  RT  trials  were  associated  with  smaller  amplitude  TT  saccades. 
Figures  4  and  5  reveal  that  the  long  RT  trials  were  associated  with  longer  duration  and  larger  latency  TT  saccades  than 
either  the  medium  or  short  RT  trials. 


3.3.2  Segregation  by  gaze  control  measures 

Extreme  trial  groups  based  upon  gaze  control  data  were  constructed  in  a  manner  analogous  to  that  just  described  for  the 
RT  distribution.  Thus,  RT  differences  were  examined  as  a  function  of  small,  medium  and  large  amplitude  TT  saccades; 
short,  medium  and  long  duration  TT  saccades;  and  short,  medium  and  long  latency  TT  saccades.  Figure  6  displays  the 
simple  RTs  associated  with  trials  differing  in  the  amplitude  of  the  associated  TT  saccade.  Although  the  main  effect  of 
amplitude  group  on  RT  just  failed  to  reach  significance  [F(2,22)=3.194;  p=0.061],  pairwise  comparisons  of  the  group 
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Figure  6:  RT  by  Saccade  Amplitude 
*p<0.05 


Figure  7:  RT  by  Saccade  Duration 

**p<0.01 


Figure  8:  RT-by  Saccade  Latency 

**p<0.01 


means  indicated  that  the  RTs  for  trials  from  large  amplitude  saccades  were  slower  than  the  RTs  associated  with  medium 
amplitude  saccades.  Figures  7  and  8  display  comparable  RT  data  for  the  groups  segregated  by  the  duration  and  latency 
measures.  Main  effects  of  both  the  duration  [F(2,22)=9.416;  p<0.001]  and  latency  [F(2,22)=  14.929;  p<0.001] 
segregations  were  obtained.  Long  duration  TT  saccades  were  associated  with  increased  RTs  (as  compared  to  both  the 
short  and  medium  duration  groups).  Similarly,  RTs  on  trials  classified  as  belonging  to  the  short  and  medium  latency  TT 
saccade  groups  were  significantly  shorter  than  when  the  trial  belonged  to  the  long  latency  classification. 
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4.  CONCLUSIONS 

A  number  of  conclusions  can  be  drawn  from  the  data  presented  above.  Sixty  min  of  continuous  performance  on  the 
EPVT  produced  significant  periods  of  performance  drop-outs  as  well  as  associated  increases  in  the  variability  of 
measures  of  gaze  control.  These  performance  lapses  tended  to  occur  with  increasing  frequency  during  later  task 
segments  resulting  in  a  number  of  TOT  effects  on  both  the  performance  and  gaze  control  measures.  Thus,  RT 
variability  increased  with  increasing  TOT  while  TT  saccade  durations  lengthened  and  became  more  variable  as  a 
function  of  TOT.  TT  saccade  amplitude  variability  also  increased  during  later  time  intervals.  Not  surprisingly, 
aggregating  data  across  20  min  segments  proved  too  coarse  an  analysis  interval  for  the  detection  of  significant 
differences  in  TT  saccade  latency.  However,  the  analysis  of  data  collapsed  across  consecutive  5  min  intervals  revealed 
that  TT  saccade  latency  was  highly  predictive  of  local  RT  performance.  The  latency  measure  was  the  best  single 
predictor  of  RT.  Notably,  for  the  two  subjects  who  evidenced  no  relationship  between  the  latency  and  RT  measures, 
substantial  correlations  between  RT  and  either  the  amplitude  or  latency  measure  were  obtained.  These  findings 
establish  the  utility  of  the  EPVT  as  a  laboratory  test-bed  for  the  proposed  real-time  alertness  monitoring  system. 
However,  the  ultimate  success  of  such  a  real-time  system  depends  upon  the  existence  of  an  even  closer  relationship 
between  performance  drop-outs  and  the  control  of  gaze.  Importantly,  significant  relationships  between  performance 
and  gaze  control  measures  were  also  obtained  for  groups  of  trials  identified  as  extreme  -irrespective  of  their  time  of 
occurrence  during  the  performance  of  the  task.  Thus,  trials  with  extremely  long  RTs  were  also  characterized  by 
increases  in  the  duration  and  latency  of  the  associated  TT  saccade.  Conversely,  increases  in  the  amplitude,  duration  and 
latency  of  TT  saccades  were  associated  with  increased  RTs  as  well.  Thus,  decreased  performance  levels  were  observed 
for  trials  characterized  by  aberrant  measures  of  gaze  control  behavior.  These  findings  are  critical  to  the  development  of 
valid  and  reliable  real-time  monitoring  systems  and  provide  a  compelling  argument  for  including  TT  saccade  metrics  in 
oculomotor-based  alertness  assessment  batteries  whenever  such  measures  are  available. 
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